1 May 1998 Source: http://www.access.gpo.gov/su_docs/aces/aaces002.html ------------------------------------------------------------------------- Joint Military Operations: Weaknesses in DOD's Process for Certifying C4I Systems' Interoperability (Letter Report, 03/13/98, GAO/NSIAD-98-73). Pursuant to a congressional request, GAO reviewed: (1) whether Department of Defense (DOD) organizations are complying with interoperability testing and certification requirements for command, control, communications, computers, and intelligence (C4I) systems; and (2) what actions, if any, are needed to improve the current certification process. GAO noted that: (1) DOD does not have an effective process for certifying existing, newly developed, and modified C4I systems for interoperability; (2) many C4I systems have not been certified for interoperability and, in fact, DOD does not know how many require certification; (3) improvements to the certification process are needed to provide DOD better assurance that C4I systems critical to effective joint operations are tested and certified for interoperability; (4) DOD organizations are not complying with the current interoperability testing and certification process for existing, newly developed, and modified C4I systems; (5) according to Test Command officials, many C4I systems that require interoperability testing have not been certified or have not received a waiver from the requirement; (6) the extent of this noncompliance could have far-reaching effects on the use of such systems in joint operations; (7) noncompliance with interoperability testing and certification stems from weaknesses in the certification process itself; (8) while DOD guidance requires that all new systems be certified or obtain a waiver from certification testing before they enter production and fielding, systems proceed to these latter acquisition stages without being certified; (9) this occurs, in part, because Defense Information Systems Agency (DISA) Joint Interoperability Test Command officials lack the authority to compel DOD organizations to submit their C4I systems for testing; (10) although DOD guidance spells out a specific interoperability certification requirement, many DOD organizations are unaware of it; (11) others simply ignore the requirement because it is not strictly enforced or because they do not adequately budget for such testing; (12) another fundamental weakness in the process is the lack of a complete and accurate listing of C4I systems requiring certification and a plan to prioritize systems for testing; (13) as a result, the Test Command may not be focusing its limited resources on certifying the most critical systems first; (14) prioritization is important since the Command has reviewed only about 100 systems per year, and a requirement for recertification of modified systems continually adds to the number of systems requiring certification; and (15) the process does not include notifying the services about interoperability problems, and the Test Command has only recently begun to contact the services regarding the noted problems. --------------------------- Indexing Terms ----------------------------- REPORTNUM: NSIAD-98-73 TITLE: Joint Military Operations: Weaknesses in DOD's Process for Certifying C4I Systems' Interoperability DATE: 03/13/98 SUBJECT: Military communication Noncompliance Testing Command/control/communications/computer systems Management information systems Waivers IDENTIFIER: DOD Command, Control, Communications, and Computer Systems Defense Integration Support Tools Data Base ****************************************************************** ** This file contains an ASCII representation of the text of a ** ** GAO report. Delineations within the text indicating chapter ** ** titles, headings, and bullets are preserved. Major ** ** divisions and subdivisions of the text, such as Chapters, ** ** Sections, and Appendixes, are identified by double and ** ** single lines. The numbers on the right end of these lines ** ** indicate the position of each of the subsections in the ** ** document outline. These numbers do NOT correspond with the ** ** page numbers of the printed product. ** ** ** ** No attempt has been made to display graphic images, although ** ** figure captions are reproduced. Tables are included, but ** ** may not resemble those in the printed version. ** ** ** ** Please see the PDF (Portable Document Format) file, when ** ** available, for a complete electronic file of the printed ** ** document's contents. ** ** ** ** A printed copy of this report may be obtained from the GAO ** ** Document Distribution Center. For further details, please ** ** send an e-mail message to: ** ** ** ** ** ** ** ** with the message 'info' in the body. ** ****************************************************************** Cover ================================================================ COVER Report to the Chairman, Subcommittee on Military Research and Development, Committee on National Security, House of Representatives March 1998 JOINT MILITARY OPERATIONS - WEAKNESSES IN DOD'S PROCESS FOR CERTIFYING C4I SYSTEMS' INTEROPERABILITY GAO/NSIAD-98-73 C4I Interoperability (703179) Abbreviations =============================================================== ABBREV C4I - command, control, communications, computers, and intelligence DISA - Defense Information Systems Agency DOD - Department of Defense Letter =============================================================== LETTER B-279021 March 13, 1998 The Honorable Curt Weldon Chairman, Subcommittee on Military Research and Development Committee on National Security House of Representatives Dear Mr. Chairman: Command, control, communications, computers, and intelligence (C4I) systems relay critical information to U.S. forces during joint operations. If joint operations are to be successful, C4I systems must be "interoperable"--capable of exchanging information and operating effectively together. To help ensure interoperability, the Defense Information Systems Agency (DISA)--under the direction of the Joint Chiefs of Staff--established the current certification process in 1992. According to Joint Staff guidance, commanders in chief, the four services, and Department of Defense (DOD) agencies are required to use this process to test and certify existing and newly developed systems for interoperability. Generally, newly developed systems are to be denied production approval if they have not been certified. After a system has been fielded and a modification is made that affects interoperability, the system must be recertified. In response to your request, we determined (1) whether DOD organizations are complying with interoperability testing and certification requirements and (2) what actions, if any, are needed to improve the current certification process. We also identified initiatives that affect interoperability; they are discussed in appendix I. BACKGROUND ------------------------------------------------------------ Letter :1 The military services have a long history of interoperability problems during joint operations. For example, the success of the Persian Gulf war in 1991--a major joint military operation--was hampered by a lack of basic interoperability. The current certification requirement was established to help address these problems. The Joint Staff's Director for C4 systems (J-6) is assigned primary responsibility for ensuring compliance with the certification requirement. DISA's Joint Interoperability Test Command is the sole certifier of C4I systems. According to Joint Staff guidance, commanders in chief, the services, and DOD agencies are required to adequately budget for certification testing. They can either administer their own tests with Test Command oversight or ask the Test Command to administer them. Certification is intended to help provide the warfighter with C4I systems that are interoperable and to enable forces to exchange information effectively during a joint mission. Specifically, certification by the Test Command is confirmation that (1) a C4I system has undergone appropriate testing, (2) the applicable requirements for interoperability have been met, and (3) the system is ready for joint use. However, while a system may pass certification testing, it may not have been tested against all systems with which it may eventually interoperate. This is because some systems with which they must interoperate become available later and commanders sometimes use systems in new ways that were not envisioned during testing. DOD guidance requires that a system be tested and certified before approval to produce and field it. Depending on the acquisition category and dollar threshold of the program,\1 the approval authority may be the Under Secretary of Defense (Acquisition and Technology), with advice from the Defense Acquisition Board; the Assistant Secretary of Defense (Command, Control, Communications, and Intelligence), with advice from the Major Automated Information System Review Council; or the DOD component head (such as the commander in chief of a unified combatant command, the head of a military service, or a DOD agency head). A DOD Directive established the Military Communications Electronics Board to provide guidance on interoperability issues referred to it by the Secretary of Defense and the Chairman of the Joint Chiefs of Staff. The Board addresses interoperability issues through two subpanels: (1) The Interoperability Improvement Panel monitors C4I interoperability issues surfaced by the commanders in chiefs, military services, and DOD agencies and (2) The Interoperability Test Panel resolves testing disputes (such as appeals of Test Command certification decisions made by commanders in chief, military services, and DOD agencies). The Test Panel may waive the certification requirement to support developmental efforts, demonstrations, exercises, or normal operations. The waiver is not intended to be permanent, and is typically granted for 1 year. -------------------- \1 DOD has four traditional acquisition categories--major defense programs, major automated system programs, other major programs, and nonmajor acquisition programs. RESULTS IN BRIEF ------------------------------------------------------------ Letter :2 DOD does not have an effective process for certifying existing, newly developed, and modified C4I systems for interoperability. As a result, many C4I systems have not been certified for interoperability and, in fact, DOD does not know how many require certification. Improvements to the certification process are needed to provide DOD better assurance that C4I systems critical to effective joint operations are tested and certified for interoperability. DOD organizations are not complying with the current interoperability testing and certification process for existing, newly developed, and modified C4I systems. According to Test Command officials, many C4I systems that require interoperability testing have not been certified or have not received a waiver from the requirement. The extent of this noncompliance could have far-reaching effects on the use of such systems in joint operations. For example, a modified C4I system that was not recertified experienced an interoperability problem exchanging data with another system. The result was the simulated downing of a commercial airplane during a joint exercise. Noncompliance with interoperability testing and certification stems from weaknesses in the certification process itself. While DOD guidance requires that all new systems be certified or obtain a waiver from certification testing before they enter production and fielding, systems proceed to these latter acquisition stages without being certified. This occurs, in part, because Test Command officials lack the authority to compel DOD organizations to submit their C4I systems for testing. Although DOD guidance spells out a specific interoperability certification requirement, many DOD organizations are unaware of it. Others simply ignore the requirement because it is not strictly enforced or because they do not adequately budget for such testing. Another fundamental weakness in the process is the lack of a complete and accurate listing of C4I systems requiring certification and a plan to prioritize systems for testing. As a result, the Test Command may not be focusing its limited resources on certifying the most critical systems first. Prioritization is important since the Command has reviewed only about 100 systems per year, and a requirement for recertification of modified systems continually adds to the number of systems requiring certification. The process also does not include a mechanism to notify the services about interoperability problems identified in joint exercises, and the Test Command has only recently begun to contact the services regarding the noted problems. Finally, the Test Panel does not have a formal process to inform DOD organizations that systems with expired waivers require an extension or certification. Accordingly, six of nine systems with expired waivers have not had the waiver extended or been tested and certified. COMPLIANCE WITH CERTIFICATION REQUIREMENT IS INADEQUATE ------------------------------------------------------------ Letter :3 Commanders in chief, services, and DOD agencies are generally not complying with the certification requirement. As a result, we found instances in which existing, newly fielded, and modified systems are not certified for interoperability. Test Command analysis showed that a significant number of existing C4I systems had not been submitted for certification as required. According to Test Command officials, as of December 1997, the DOD Defense Integration Support Tool database of C4 systems listed about 1,000 systems that may exchange information with another system. In addition, there are about 1,176 unclassified intelligence systems, according to the Office of the Assistant Secretary of Defense, C3I. Test Command officials said they did not know precisely how many of these systems require certification. Nor did the Office of the Assistant Secretary of Defense know which intelligence systems would require certification because they were unable to determine which of these systems were outdated (i.e., legacy systems), stand alone systems, or one-service-only systems. While the Test Command has generally certified increasingly more systems during the past 4 years, officials acknowledged that "they have not even begun to scratch the surface" of the universe of systems that may require testing and certification. During fiscal years 1994 through 1997, the Test Command certified 149 C4I systems. According to Test Command officials, DOD's Defense Integration Support Tool database attempts to list all C4 systems and other mission critical systems, but it does not contain all C4 systems or indicate whether the systems have been certified. According to DISA documentation, the purpose of the Defense Integration Support Tool is to support a DOD-wide information management requirement for data collection, reporting, and decision support in areas such as planning and interoperability. After discussions with DOD officials regarding this issue, DOD has recently included certification status as part of the database and, as of January 1998, 44 systems reflected this information. We recently reported in two separate reports that the Defense Integration Support Tool database is incomplete and inaccurate.\2 In response to our October 1997 report, DOD acknowledged that this database is its official automated repository and backbone management tool for DOD's inventory of systems. Accordingly, DOD said that it had begun to take major actions to enhance the database by instituting a validation and data quality program to ensure that the database contains accurate and complete data. DOD further stated that it would closely monitor this program to ensure that the data quality is at the highest level as required for reports to senior Defense managers and the Congress. Since this database is an important management tool, it is essential that it be complete and accurate. In several instances, new systems have been fielded without consideration of the certification requirement. Two recently fielded Air Force systems--a weather prediction system and a radar system--were not tested for certification by the Test Command, despite June 1996 memorandums from the Joint Staff stating that the service must plan for testing to ensure compliance with interoperability guidelines. Further, since 1994, the Assistant Secretary of Defense (Command, Control, Communications, and Intelligence) has approved three of nine major automated information systems for production and fielding that had not been certified for interoperability. For example, the recently fielded Defense Message System was not certified by the Test Command. Test Command officials stated that the system has undergone some interoperability testing but, because of shortfalls, was not certified. A decision was made to field the system while the shortfalls are resolved. Test Command officials believe the system will eventually be certified. No newly developed systems purchased through the Command and Control Initiatives Program were tested by the Test Command. (This program allows commanders in chief to purchase low-cost improvements to their command and control systems.) According to DISA officials, DISA had assessed these systems' interoperability requirements and reminded the users to submit the systems for testing. In addition, during the last 3 years, no systems purchased through the Advanced Concept Technology Demonstrators program were tested and certified. (This program allows a new capability to be quickly developed, purchased, and exercised in the field before an acquisition commitment is made.) According to Test Command officials, previously certified systems that were later modified are not consistently submitted for recertification as required. Although Test Command officials do not know the exact number of modified systems that require recertification, they are aware of several systems--such as the Navy's AEGIS shipboard weapon system and the Air Force's Airborne Warning and Control System. -------------------- \2 Defense Computers: Improvements to DOD Systems Inventory Needed for Year 2000 Effort (GAO/AIMD-97-112, Aug. 13, 1997) and Defense IRM: Poor Implementation of Management Controls Has Put Migration Strategy at Risk (GAO/AIMD-98-5, Oct. 20, 1997). REASONS FOR INADEQUATE COMPLIANCE ---------------------------------------------------------- Letter :3.1 Joint Staff officials believe that, although the certification requirement is outlined in several DOD and Joint Staff guidance documents, some system managers are unaware of it.\3 In a study chartered by J-6 and completed in January 1996, only 12 of 424 (less than 3 percent) surveyed acquisition managers and Defense System Management College students knew about the DOD and Joint Staff interoperability requirements. The study team found that this lack of knowledge prevented users from placing interoperability in the initial requirements documents and acquisition managers from building interoperability into approved programs. As a result, the Joint Staff began an effort in 1996 to better educate system managers about the requirement. However, the study points out that education is not a panacea for all interoperability problems. Our analysis showed that some DOD organizations, although aware of the requirement, did not submit fielded systems for testing. For example, some program managers did not submit their modified systems for certification because they believed their design, although fielded, was not mature enough for testing. The program managers did not seek a waiver for their systems and ignored the certification requirement. Test Command officials told us that they lack the authority to compel program managers to bring their systems in for testing and must rely on the managers' cooperation. In addition, in fiscal year 1995, only three intelligence systems were certified by the Test Command. Because Test Command officials believed that DOD's intelligence community was ignoring the certification requirement, in 1996 the Command negotiated an agreement with DOD's Intelligence Information Systems Management Board (which has responsibility for a portion of intelligence systems) to facilitate better participation in the certification process. In fiscal year 1997, the number of intelligence systems tested and certified increased to 14. Test Command officials believe that the increase is a direct result of the agreement. Further, according to Test Command officials, DOD officials do not always budget the resources needed for interoperability testing as required by Joint Staff guidance. In certain cases, the services do not budget sufficient funds to cover secondary C4I systems that are used to test the primary C4I system for interoperability because the services cannot afford to pay for all the testing DOD policy requires. For example, the services are required to provide secondary systems for 10 tactical data link interoperability tests a year. In this case, however, according to a Test Command official, the Army budgets for only seven or eight tests a year. The services are responsible for acquiring systems that satisfy service-unique requirements, and this responsibility sometimes takes precedence over satisfying joint interoperability requirements. In his 1996 report to the Secretary of Defense, the Chairman of the Joint Chiefs of Staff recommended that funding for DOD C4I systems be reviewed, since the services' funding decisions may not further DOD's overall goal of promoting C4I joint interoperability. Finally, the various approval authorities are allowing some new systems to be fielded without verifying their certification status. According to a Joint Staff J-6 spokesman, the Joint Staff J-6 representative is to ensure that interoperability certification is addressed at the approval authority acquisition meetings. If the Joint Staff J-6 representative is unable to attend these meetings, the issue of certification is not raised. However, J-6 coordination is obtained on all acquisition decision memorandums granting production and fielding approval. Nevertheless, systems receive approval for production and fielding even though they may not have been certified or obtained waivers. -------------------- \3 The primary DOD interoperability guidance documents are DOD Directive 4630.5, November 12, 1992; DOD Instruction 4630.8, November 18, 1992; and Chairman of the Joint Chiefs of Staff Instruction 6212.01A, June 30, 1995. EXAMPLES OF INTEROPERABILITY PROBLEMS THAT ARE NOT BEING ADDRESSED ---------------------------------------------------------- Letter :3.2 In several instances, the Test Command identified interoperability problems in systems that DOD organizations had not submitted for testing. The following are examples: In 1996, the Test Command expressed concerns to the Air Force that its Joint Tactical Information Distribution System, a computer terminal used to provide surveillance data on F-15 aircraft, had not been certified. The system (a proof of concept demonstration) had operated for 3 years. According to a Test Command memorandum, Command representatives witnessed numerous interoperability problems caused by this system during joint exercises. The memorandum indicated that if the exercise had been a real world situation, the system's interoperability problems could have resulted in numerous deaths of pilots and enemy penetrations of U.S. airspace. In a written response, the Air Force stated that it disagreed with the Test Command's assessment of the problems. Furthermore, the Air Force said that certification of the system was not the best use of resources because the Air Force planned to eventually replace it. According to Test Command officials, the system is scheduled for testing in 1998. Still not certified, the system has been operational for over 1 year since the Air Force's response. Test Command officials have been unable to persuade the Navy's AEGIS program office to submit all fielded versions of the ship's weapon system for interoperability testing. Command representatives have observed the weapon system experiencing significant interoperability problems in several recent joint exercises. The Test Command is aware of five fielded versions of AEGIS software, and the program office states there are many more. However, the Test Command has tested and certified only the oldest version (in May 1995), the most basic of the five versions. The need for interoperability certification testing of the uncertified versions has been discussed at joint interoperability meetings and with DISA. The responsible DISA official requested, under Test Command letterhead, that AEGIS submit uncertified versions for joint testing. However, according to AEGIS program officials, none of these versions has been jointly tested because the newer versions either have not yet been tested with other Navy-only systems or are not yet demonstrating adequate interoperability performance in testing with Navy-only systems. The Test Command has been unable to persuade users to test DOD's Air Defense System Integrator, which provides tactical data link translation and message-forwarding functions. The system has been acquired outside the normal DOD acquisition process. About 30 versions of this system have been fielded; none has been jointly tested. According to Test Command officials, the system is experiencing significant interoperability problems because it does not conform to required standards. Interoperability problems with this system could result in hostile systems leaking through U.S. defenses or friendly systems being attacked. Without certification of the interfaces that translate and forward messages among systems, for example, the proper tracking and targeting information may not be provided to our theater air missile defense system. At several 1997 meetings with representatives from all the services, the Joint Staff, and the Test Command, problems with the system were discussed. Solutions are still being developed and implemented. WEAKNESSES EXIST IN DOD'S CERTIFICATION PROCESS ------------------------------------------------------------ Letter :4 Noncompliance with interoperability testing and certification stems from weaknesses in the certification process itself. For example, DOD lacks a complete and accurate listing of C4I systems requiring certification and a plan to prioritize systems for testing. As a result, the Test Command may not be focusing its limited resources on certifying the most critical systems first. The process also does not include a mechanism to notify the services about interoperability problems identified in joint exercises, and the Test Command has only recently begun to contact the services regarding the noted problems. Finally, according to a Test Panel official, the Panel does not have a formal process to inform DOD organizations that systems with expired waivers require an extension or certification. DOD LACKS A PLAN TO PRIORITIZE TESTING AND HAS NOT IDENTIFIED CRITICAL SYSTEMS TO BE CERTIFIED ---------------------------------------------------------- Letter :4.1 Neither the Joint Staff nor DISA has given the Test Command a priority list for testing C4I systems. As a result, the Command tests systems without regard to systems that should receive a high priority for testing. Test Command officials believe that such a list would help them better plan their test schedule. Generally, the Command develops a master test schedule based on the notification of systems ready for testing by the commanders in chiefs, services, and DOD agencies. As these notifications are received, the Command updates its schedule. Furthermore, DOD has not identified the exact number of systems to be certified. A Command official told us that, even if systems are identified, it is difficult to test all C4I systems required to be certified. According to Test Command officials, they are able to test no more than 200 systems per year. Our analysis shows that the Command generally reviews about 100 systems per year and in 1997 certified 44 individual systems for interoperability (not including systems receiving multiple certifications due to modifications or testing with additional systems). According to the official, a list prioritizing systems for testing would assist the Command to use its scarce resources to test the most important systems first. In June 1996, the Military Communications Electronic Board reviewed existing command and control systems submitted by the services and determined that 42 were crucial to the needs of military commanders. Our analysis showed that, as of October 1997, 23 had not been tested or certified. According to Test Command officials, the 23 systems were not certified for various reasons. The officials stated that they did not know about 13 of the systems; 7 are scheduled or are to be scheduled for testing, but the schedules could slip; 2 were not submitted for testing by the commanders in chief, service, or DOD agency because 1 is a low priority for testing and the other needs redesign (although both have been operational for several years); and 1 was considered too immature to test. Without an approved DOD-wide testing strategy, the Test Command's scarce resources may not be best used to test the right C4I systems at the right time. Joint Staff, Test Command, and commander in chief officials believe that one area that should receive high priority in any plan for interoperability testing is theater air and missile defense systems. This functional area is heavily dependent on systems being interoperable. According to Test Command officials, about 100 major systems are involved in theater air and missile defense, and about 45 percent of these have not been tested or certified for interoperability. DOD officials stated that significant interoperability problems in these defense systems could have dire consequences for joint and coalition forces. Some joint exercises conducted during the last 2 years have demonstrated the need for better interoperability in this functional area. Interoperability problems in these exercises resulted in the simulated downing of friendly aircraft in one exercise and in the nonengagement of hostile systems in another. TEST COMMAND DOES NOT ADVISE THE SERVICES ABOUT INTEROPERABILITY PROBLEMS ---------------------------------------------------------- Letter :4.2 Test Command officials stated that they do not generally advise services' system program managers on interoperability problems identified in exercises. While not required to do so, the Test Command is in the best position to advise the commanders in chief, services, and DOD agencies because according to Command officials they discover, evaluate, and document these problems. As part of its mission and apart from certification testing, the Command provides operational support and technical assistance to the commanders in chief, the services, and DOD agencies during exercises. In reports summarizing the results of four joint exercises during 1996 and 1997, the Test Command noted that 15 systems experienced 43 �significant interoperability problems�--defects that could result in the loss of life, equipment, or supplies. The vast majority of these problems were caused by system-specific software problems. Specific problems experienced included -- failure to accept changes in mislabeled data identifying a friendly aircraft as a hostile aircraft, thereby causing the simulated downing of a commercial airliner; -- excess messages overloading systems, causing system crashes and the loss of command and control resources during critical periods; -- improper track identification, creating the potential for either a hostile system to penetrate defenses or a friendly system to be inadvertently destroyed; and -- duplicate tracks distorting the joint tactical picture, denying vital information to battle managers and shooters. In table 1, we list the 15 systems that experienced significant problems and indicate their certification status. Table 1 Certification Status of C4 Systems Experiencing Significant Interoperability Problems in Four Joint Exercises During 1996 and 1997 Certification status ---------------------------------------- Number of significant Modified but interoperability not C4 systen problems\ Certified\a Uncertified recertified -------------------- ---------------- ------------ ------------ ------------ PATRIOT 8 X AEGIS 7 X Shelterized Joint 6 X Tactical Information Distribution System Modular Control 4 X Equipment Airborne Warning and 3 X Control System Airborne 3 X Surveillance Testbed Tactical Air 2 X Operations Module Joint Tactical 2 X Ground Station Air Defense System 2 X Integrator EP-3E ARIES aircraft 1 X F-15 aircraft 1 X F-14 aircraft 1 X Airborne Laser\b 1 X Theater High- 1 X Altitude Air Defense\c Expert Missile 1 X Tracker ================================================================================ Total 43 -------------------------------------------------------------------------------- \a Even though a system is certified, significant problems not identified during testing can arise in an exercise due to the less-controlled environment. Also, systems used in exercises often are linked by radio rather than direct cable connection, introducing the potential for missing information. Other problems with certified systems could surface because new systems with which they must interoperate might not have been in the force when testing occurred. Also, commanders sometimes use systems in ways not envisioned during testing. \b This system has not yet been approved for production and fielding and has not been tested for interoperability by the Test Command. \c Some components of this system participated in a joint exercise. Interoperability testing is scheduled to begin in March 1999. Source: Our analysis of 1996 and 1997 Joint Interoperability Test Command exercise reports. When the services' program managers are not advised, significant interoperability problems may arise in subsequent exercises and operations. According to Test Command officials, after our inquiries the Command began exploring ways to formally track and follow up on these problems. After our visit, Command officials stated they were beginning to identify the problem systems and contact the program managers to request that systems be retested. However, as of December 1997, Command officials had contacted only three system managers, and none of the systems have been tested. TEST PANEL DOES NOT HAVE A FORMAL PROCESS FOR INFORMING DOD ORGANIZATIONS ABOUT EXPIRED WAIVERS ---------------------------------------------------------- Letter :4.3 According to a Test Panel official, the Panel does not have a formal process to ensure that fielded systems with expired waivers are tested. As a result, most systems with expired waivers were allowed to operate without testing or an extension of the waiver. According to Panel documents, 13 waivers have been granted since May 1994. Of the 13 waivers granted, 3 have not expired and 1 was recently extended after the original waiver had been expired for 4 months (even though the system has caused interoperability problems). The remaining nine waivers have expired. Of these nine, only three are for systems that have had some interoperability testing and certification by the Test Command. Of the remaining six systems with expired waivers, two were expired for less than a year, two were expired for more than a year, and two were expired for more than 2 years. CONCLUSIONS ------------------------------------------------------------ Letter :5 Commanders in chief, the services, and DOD agencies are generally not complying with the C4I certification requirement. Inadequate compliance with this requirement increases the likelihood that C4I systems will not be interoperable, thereby putting lives, expensive equipment, and the success of joint military operations at greater risk. Improvements to the certification process are needed to provide better assurance that C4I systems most critical to joint operations are certified for interoperability. Better information is needed to track the status of waivers. Finally, the risks associated with operating uncertified systems in joint operations is heightened when systems are permitted to proceed into production and fielding without full consideration of the certification requirement. RECOMMENDATIONS ------------------------------------------------------------ Letter :6 To ensure that systems critical to effective joint operations do not proceed to production without due consideration given to the need for interoperability certification, we recommend that the Secretary of Defense require the acquisition authorities to adhere to the requirement that C4I systems be tested and certified for interoperability prior to the production and fielding decision unless an official waiver has been granted. To improve the process for certifying C4I systems for interoperability, we recommend that the Secretary of Defense, in consultation with the Chairman of the Joint Chiefs of Staff, direct the -- service secretaries, in collaboration with the Director of DISA to verify and validate all C4 data in the Defense Integration Support Tool and develop a complete and accurate list of C4I systems requiring certification and -- Director of DISA to ensure that the status of system's certification is added to the Defense Integration Support Tool and that this database be properly maintained to better monitor C4 systems for interoperability compliance. We also recommend that the Secretary of Defense request that the Chairman of the Joint Chiefs of Staff direct the -- Joint Staff (in collaboration with the commanders in chief, the services, and the Director of DISA) to develop a process for prioritizing C4I systems for testing and certification and -- Joint Staff (in collaboration with the commanders in chief, the services, and the Director of DISA) to develop a formal process to follow up on interoperability problems observed during exercises, report the problems to the relevant DOD organization, and inform organizations that the systems are required to be tested for interoperability. We recommend that, to improve DOD's information on the status of waivers from interoperability certification, the Chairman of the Joint Chiefs of Staff establish a system to monitor waivers. The system should inform DOD organizations when waivers expire and request that they either seek an extension of the waivers or test their systems for interoperability. AGENCY COMMENTS ------------------------------------------------------------ Letter :7 In written comments on a draft of this report, DOD generally concurred with all of our recommendations noting that a number of efforts are underway to improve the interoperability certification process. To improve the process, DOD is revising relevant policy and procedures to enhance their adequacy (in terms of clarity, enforcement, and integration of effort) and is improving the accuracy and utility of its Defense Integration Support Tool database. Agreeing with the need to prioritize systems for testing, DOD stated it will develop a process to set priorities for testing and certification. To follow up on interoperability issues learned during exercises, DOD intends to use several sources of information to develop a formal process to ensure identified problems are adequately addressed by the appropriate organizations. DOD also intends to revise the charter of the Test Panel to require quarterly review of waivers from certification testing. DOD's comments are reprinted in appendix II. DOD also provided technical comments, which we have incorporated where appropriate. SCOPE AND METHODOLOGY ------------------------------------------------------------ Letter :8 To determine whether DOD organizations were complying with the certification requirement, we analyzed DOD data on C4I systems to identify systems' certification status. Specifically, we obtained a listing of all C4 systems in the Defense Integration Support Tool from DISA Headquarters in Arlington, Virginia, and the number of unclassified intelligence systems from the Office of the Assistant Secretary of Defense, C3I in Arlington, Virginia. We compared the systems on these lists with a list of all systems certified from October 1993 through September 1997 obtained from the Joint Interoperability Test Command in Fort Huachuca, Arizona. We also obtained a list of C4I systems included in Command and Control Initiatives Program budget proposals from October 1994 through September 1997 and a listing of C4I systems included in DOD's Advanced Concept Technology Demonstrators program. We compared these lists with the Test Command's list of certified systems. We did not verify the accuracy or validity of any DOD list. We also obtained, reviewed, and analyzed DOD policy, Joint Staff instructions, and other documents regarding compatibility, interoperability, and integration of C4I systems. We obtained these documents and discussed interoperability issues in the Washington, D.C., area in interviews with cognizant officials from the Office of the Deputy Under Secretary of Defense (Advanced Technology); the Office of the Assistant Secretary of Defense, C3I; the Office of the Director, Operational Test and Evaluation; the Joint Chiefs of Staff Directorate for C4 (J-6); the Directorate for Force Structure, Resources and Assessment (J-8); and DISA. In addition, we reviewed documents and interviewed cognizant officials regarding interoperability issues, including certification of C4I systems, from the U.S. Atlantic Command, Norfolk, Virginia; U.S. Central Command, MacDill Air Force Base, Florida; U.S. Pacific Command, Camp Smith, Hawaii; U.S. European Command, Germany; the Naval Center for Tactical Systems Interoperability, San Diego, California; U. S. Army Communications and Electronics Command, Fort Monmouth, New Jersey; and individual system program offices or support activities in each of the military services, including the Navy AEGIS program office, Dahlgren, Virginia; the Air Force Air Combat Command Directorate of Operations for Command and Control and Intelligence, Surveillance, and Reconnaissance, Langley Air Force Base, Virginia; the Army Communications and Electronics Command Software Engineering Center, Fort Monmouth, New Jersey; and the Naval Air Warfare Center, Weapons Division, Point Mugu, California. To determine whether improvements were needed in the certification process, we interviewed Test Command officials on interoperability and certification issues, including testing priorities and exercise problem follow-up, and compared the Command's list of certified systems from October 1993 through September 1997 with a June 14, 1996, list of DOD's crucial C2 systems. We also reviewed reports on lessons learned and demonstrations and exercises obtained from the Joint Staff J-8 and the Test Command, respectively, to identify C4I systems with interoperability problems. We then compared the problem C4I systems with the Test Command's certification list to analyze whether the systems were certified, uncertified, or modified and not recertified. We also interviewed officials and obtained and analyzed waiver documents from the Military Communications Electronics Board's Interoperability Test Panel. We reviewed the waivers to determine the reasons for them and the time period involved. Finally, to determine initiatives that affect interoperability, we reviewed DOD's C4I for the Warrior concept; the Defense Information Infrastructure Master Plan; the 1996 assessment of combat support agencies report by the Chairman of the Joint Chiefs of Staff; the 1996 Command, Control, Communications, Computer, Intelligence, Surveillance, and Reconnaissance Task Force reports; and the Levels of Information System Interoperability reports by the Task Force. We conducted our review from January 1997 to January 1998 in accordance with generally accepted government auditing standards. ---------------------------------------------------------- Letter :8.1 We are sending copies of this report to the Secretaries of Defense, the Army, the Navy, and the Air Force and other appropriate congressional committees. Copies will also be made available to others on request. Please contact me at (202) 512-5140 if you or your staff have any questions concerning this report. Major contributors to this report are listed in appendix III. Sincerely yours, Mark E. Gebicke Director, Military Operations and Capabilities Issues DOD INITIATIVES TO IMPROVE THE INTEROPERABILITY OF C4I SYSTEMS =========================================================== Appendix I Improving ways of complying with the certification process alone will not solve all of the issues related to interoperability. The Department of Defense (DOD) has a number of initiatives underway that address various aspects of interoperability: the C4I for the warrior concept; the Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance Architecture Framework; the Defense Information Infrastructure strategy; and the Levels of Information Systems Interoperability initiative. Initiated in 1992, the C4I for the warrior concept is to provide a global command, control, communications, computer, and intelligence system that directly links and supports the combat troops of all services who engage in military operations. The system will display anywhere around the world a real-time, true picture of the battlespace, detailed mission objectives, and a clear view of enemy targets. This advanced technology concept is to support DOD's vision for the evolution of the U.S. armed force's capabilities to the year 2010. The Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance Architecture Framework, published in June 1996 by the DOD Integration Task Force, is to address a DOD-wide lack of a shared understanding of the architecture process and insufficiently precise terminology. According to the Task Force, architectures can be a key factor in guiding and controlling the acquisition and evolution of interoperable and efficient C4I systems. If adopted, the framework will provide a common approach for the commanders in chief, the services, and DOD agencies to follow in developing their C4I architectures. The Task Force report stated that the framework has, in part, the ultimate potential of "facilitating, improving, and ensuring compatibility, interoperability, and integration among command, control, communications, computers, intelligence, surveillance, and reconnaissance capabilities." While a final report was issued in June 1996, the framework has not been implemented as DOD policy. Currently, adoption of the framework in DOD policy is not planned according to a Joint Staff official. A current version of the framework itself was issued in July 1998. However, a J-6 official expects full implementation to take 1 to 2 years after its publication. DOD issued a Defense Information Infrastructure master plan in November 1994 to integrate its communications networks, computers, software, databases, applications, weapon system interfaces, data, security services, and other services that meet DOD's information processing and transport needs. The plan is updated periodically and provides a description of the Defense Information Infrastructure's major components. The infrastructure is largely an unintegrated collection of systems with unique characteristics. These systems support a hierarchical, vertical military chain of command structure. They were not designed to support joint operations and are therefore limited when information requirements are based on horizontal or functional sources. The current infrastructure inhibits interoperability necessary to give commanders a unified picture of the battlespace, reduces ability to provide links between the battlefield and the support base, and limits connection to the U.S. industrial base. One part of the Defense Information Infrastructure plan is to establish a common operating environment that provides integrated support services and corresponding software for standard functional applications. The idea for the common operating environment originated with an observation about command and control systems. Certain functions (mapping, track management, and communication interfaces, for example) are so fundamental that they are required for virtually every command and control system. Yet, in stand-alone systems across DOD, these functions are built over and over again in incompatible ways, even when the requirements are the same or vary only sightly between systems. The common operating environment is intended to standardize the underlying computing infrastructure used to process information. It is to improve interoperability by creating architecture principles that, if adhered to, will allow for the sharing of software products and services and information across the Defense Information Infrastructure. Both the Defense Information Infrastructure plan and the common operating environment are long-term strategies that extend through the year 2010. Finally, DOD's 1993 Levels of Information Systems Interoperability initiative is to improve C4 and intelligence systems' interoperability. System developers are to use this tool to assess interoperability, determine capabilities needed to support system development, and determine the degree of interoperability needed between C4I and other systems. The tool has not yet been fully tested or implemented. Major testing is planned for July 1998. Concerns regarding the success of some of these initiatives have been expressed by various DOD organizations. Specifically, in its June 1996 report, the DOD Integration Task Force stated that compliance with the common operating environment standards will not ensure that systems will be interoperable because, in part, it does not eliminate the problems of data translation, remapping, and duplication. Further, Test Command officials and others believe the DOD Information Infrastructure and common operating environment requirements need refinement before they can ensure interoperability. For example, these officials believe that the level of compliance with the infrastructure and the common operating environment must be higher than currently required to ensure interoperability. In addition, in a December 1996 report, the Chairman of the Joint Chiefs of Staff listed several challenges to achieving interoperability through DOD's initiatives, including security of the infrastructure, overall integration of the DOD organizations into a common operating environment, and the lack of a formal enforcement mechanism to ensure the services conform to the standards. (See figure in printed edition.)Appendix II COMMENTS FROM THE DEPARTMENT OF DEFENSE =========================================================== Appendix I (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) (See figure in printed edition.) MAJOR CONTRIBUTORS TO THIS REPORT ========================================================= Appendix III NATIONAL SECURITY AND INTERNATIONAL AFFAIRS DIVISION, WASHINGTON, D.C. Carol R. Schuster Reginald L. Furr, Jr. Mae F. Jones LOS ANGELES OFFICE George Vindigni Yelena K. Thompson David G. Hubbell *** End of document. ***